APPARATUS AND PROGRAM FOR IMAGE CLASSIFICATION 

BACKGROUND OF THE INVENTION 
Field of the Invention 
5 The present invention relates to an apparatus for 

classifying image data sets obtained by a digital camera or the 
like, and to a program for causing a computer to execute 
classification processing . 

Description of the Related Art 
10 Conventional prints generated from silver salt 

photographic films have been used in various manners. For 
example, such prints are used for checking how the prints 
(photographs) look, for communication with friends by viewing 
the prints, for sending additional prints to friends, for 
15 display of enlargements thereof, and for generation of 
photograph albums. 

Meanwhile, images are often managed in the form of image 
data sets, and the image data sets are obtained by reading the 
images recorded on a film or prints, or by a digital camera. 
20 When the image data sets obtained in the above manner are managed, 
a user who manages the image data sets generates folders 
corresponding to the date of photography or events, and 
classifies the image data sets manually into the folders while 
confirming the images. 
25 However, classification of image data sets becomes more 
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troublesome as the number of the image data sets grows. 
Especially, in the case of a digital camera, a large number of 
imaged data sets are often obtained, since no cost is necessary 
for films. Therefore, various methods have been proposed for 
5 automatically classifying image data sets. For example, image 
data sets are classified according to recording media wherein 
the image data sets were recorded, or according to films from 
which the image data sets were obtained. Alternatively, image 
data sets are classified in hierarchal folders corresponding 

10 to years, months, and dates. Furthermore, in another method, 
image data sets are classified according to the date of 
photography, with reference to information representing time 
and date of photography added to the image data sets (Japanese 
Unexamined Patent Publications No. 5 (1993) -165935 and 

15 2001-228528) . In still another method, image data sets are 
classified first according to date of photography, and 
classified further according to similarity between images 
(Japanese Unexamined Patent Publication No. 2000-112997) . In 
yet another method, image data sets obtained by a digital camera 

20 are classified according to orientation of the digital camera 
at the time of photography or according to images having the 
same time information (U.S. Patent No. 5,576,759, or according 
to predetermined events related to the date of photography 
(Japanese Unexamined Patent Publication No. 2002-096145) . 

25 By using such a method of automatic image data 
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classification, a user can efficiently classify image data 
sets . 

Meanwhile, image data sets dealt with by users have been 
diversified for private use or for study, since image data sets 
5 are easy to obtain. Consequently, classification according to 
items specific to each field or according to more detailed items 
is often desired. For example, in the case of image data sets 
representing images of fish, the items to classify subjects of 
the images refer to sea fish/fresh-water fish, families they 
10 belong to, and adult fish/immature fish. In the case of image 
data sets related to recipes, the items refer to cooking time, 
cost, main ingredients, and calories, for example. However, 
according to the above-described methods that have been 
proposed, the items for classification have been predetermined 
15 and limited to general items. Therefore, the degree of freedom 
in classification is low. 

SUMMARY OF THE INVENTION 
The present invention has been conceived based on 
consideration of the above circumstances. An object of the 
2 0 present invention is therefore to provide an image 
classification apparatus enabling classification of image data 
sets with a higher degree of freedom, and a program that causes 
a computer to execute classification procedures. 

An image classification apparatus of the present 
25 invention is an apparatus for classifying image data sets added 
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with accompanying information including a plurality of 
information items, and the image classification apparatus 
comprises : 

accompanying information obtaining means for obtaining 
5 the accompanying information from the image data sets; 

information item specification means for specifying a 
portion of or all of the information items of the accompanying 
information to be used for image classification, and for 
specifying priority among the specified information items; and 
10 image classification means for classifying the image data 

sets into groups having a hierarchical structure of the 
specified information items according to the specified 
priority. 

The accompanying information refers to information added 
15 to the image data sets for representing a characteristic, a 
nature, and a state of the image data sets, for example. More 
specifically, the accompanying information represents time and 
date of photography, a photography condition, a photography 
location, a size of an image, the type of a subject, the name 
20 of the subject, the number of objects as the subject, an event, 
and a comment, for example. 

The information items refer to item names regarding the 
accompanying information. The information items include not 
only the items in the accompanying information but also items 
25 of more detailed information extracted from the accompanying 



information. For example, if one of the items in the 
accompanying information represents the time and date of 
photography, the information items can be ''photography year", 
''photography month", and "photography date". 
5 The photography condition refers to various settings of 

an imaging apparatus and weather at the time of photography. 
For example, the photography condition refers to shutter speed 
such as 1/125 second, an aperture state such as f 11, and "cloudy" 
as the weather. 

10 The size of an image refers to an aspect ratio or the number 

of pixels of the image represented by each of the image data 
sets, for example. The size can be expressed as "640X400", 
for example. 

The type of subject refers to people, mountain, sea, and 
15 building, for example. 

The name of the subject refers to a person's name such 
as "Mr. Yamada" and "Ms. Tanaka" in the case of the person being 
the subject. In the case where the subject is a mountain or 
sea, the name refers to "Mt . Takao" and "Miura Beach", for 
2 0 example. 

The number of objects as the subject refers to the number 
of people in the image as the subject thereof, in the case that 
the type of subject is people. 

The comment refers to a word or a sentence expressed by 
25 a user such as a photographer, regarding the image. For example, 
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"A walk with my dog Shiro", and "Haruo's first experience in 
a pool" can be used as the comment. 

The accompanying information can be described as tag 
information of the image data sets. 
5 In the case where the image data sets have been obtained 

by a digital camera, the accompanying information is added to 
the image data sets by the digital camera. In the case where 
the image data sets have been obtained by reading images 
recorded on a film or the like with reading means such as a 

10 scanner, the accompanying information is added to the image data 
sets by the reading means. The accompanying information may 
be added to the image data sets by the digital camera or by the 
reading means either in an automatic manner or through a manual 
operation by the user. 

15 Classifying the image data sets into groups having a 

hierarchical structure of the specified information items 
according to the specified priority refers to firstly 
classifying the image data sets according to the top-priority 
information item so that the image data sets are classified into 

20 groups at the top of the hierarchical structure, and by 
classifying the image data sets in each of the groups in the 
top hierarchical structure according to the information item 
having the second-highest priority so that the groups 
classified in this manner are positioned second highest in the 

25 hierarchical structure. In this manner, the classification is 



repeated according to the priority, and the image data sets are 
classified into the groups having the hierarchical structure . 

In an image classification apparatus of the present 
invention, the accompanying information may include 
5 classification condition information representing a set of the 
information items to be used for image classification so that 
the information item specification means can specify the 
information items to be used for image classification according 
to the classification condition information. 

10 In an image classification apparatus of the present 

invention, the accompanying information may include 
classification condition information representing a 
combination of the information items to be used for image 
classification and the priority thereof so that the information 

15 item specification means can specify the information items to 
be used for image classification and the priority thereof, 
according to the classification condition information. 

The classification condition information represents the 
set of the information items to be used for image classification, 

20 and the classification condition information is described by 
a creator or user of the image data sets or by a photography 
apparatus, for example. According to the type or category of 
the image data sets, the classification condition information 
represents the set of the information items or the combination 

25 of the information items and the priority thereof. The 
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classification condition information represents the 
combination of the information items and the priority as "1. 
photography location, 2. photography year, 3. photography month, 
4. photography date" , for example. 
5 In the image classification apparatus of the present 

invention, the information item specif ication means may specify 
the information items and the priority thereof according to a 
manual operation by a user or in an automatic manner according 
to the classification condition information included in the 

10 accompanying information. Alternatively, the user may select 
either the manual operation or the automatic manner. In 
addition, only the information items to be used for image 
classification may be determined according to the 
classification condition information so that the user can 

15 specify the priority thereof. Alternatively, the user may 
specify the information items and the priority thereof by 
modifying the combination of the information items and the 
priority represented by the classification condition 
information. Furthermore, the classification condition 

20 information may represent a plurality of combinations of the 
information items and the priority thereof so that the user can 
select one of the combinations. 

The means of the image classification apparatus may be 
provided as procedures of a program for causing a computer to 

25 execute the functions of the means. 



The image data sets may be still image data sets or video 
data sets representing moving images. In addition, the image 
data sets may include audio information representing a sound. 
According to the image classification apparatus having 
5 the configuration described above, the accompanying 
information including the information items is obtained from 
the image data sets, and the information items to be used for 
image classification are specified among the information items . 
The priority of the specified information items is also 

10 specified so that the image data sets are classified into the 
groups having the hierarchical structure of the specified 
information items according to the specified priority. 
Therefore, the image data sets can be classified with a high 
degree of freedom. 

15 If the accompanying information includes the 

classification condition information representing the set of 
the information items to be used for image classification and 
the information items are specified for image classification 
according to the classification condition information, the 

20 image data sets can be classified by the specified information 
items without specification of the information items and the 
priority thereof by the user. 

If the accompanying information includes the 
classification condition information representing the 

25 combination of the information items and the priority thereof 
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to be used for image classification, and the information items 
and the priority thereof are specified for image classification 
according to the classification condition information, the 
image data sets can be classified by the specified information 
5 items and the specif ied prior ity thereof, without specification 
of the information items and the priority thereof by the user. 
BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a block diagram showing the configuration of 
an image storage system as an image classification apparatus 
10 of a first embodiment of the present invention; 

Figure 2 shows diagrams for explaining classification of 
image data sets; 

Figure 3 shows a result of the classification of the image 
data sets; 

15 Figure 4 shows another result of the classification of 

the image data sets; 

Figure 5 is a flow chart showing procedures carried out 
in the first embodiment; 

Figure 6 shows a result of the classification of the image 
20 data sets displayed on a personal computer; and 

Figure 7 is a flow chart showing procedures carried out 
in a second embodiment of the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Hereinafter, embodiments of the present invention will 
25 be explained with reference to the accompanying drawings. 
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Figure 1 is a block diagram showing the configuration of 
an image storage system as an image classification apparatus 
of a first embodiment of the present invention. The image 
storage system shown in Figure 1 exchanges data between a 
5 personal computer 10 owned by a user 1 and an image storage server 
2 having the image classification apparatus of the present 
invention. The image storage server 2 is connected to the 
personal computer 10 of the user 1 via a network 3. 

The user 1 obtains image data sets SO by using a digital 

10 camera 11 of his/her own, and records the image data sets SO 
in a memory card 12. The user 1 reads the image data sets SO 
from the memory card 12 by using the personal computer 10, and 
stores the image data sets SO in a hard disc of the personal 
computer 10 . When the user requests classification of the image 

15 data sets SO, the user sends the image data sets SO to the image 
storage server 2 via the network 3. 

The digital camera 11 records information on time and date 
of photography (hereinafter referred to as the photography date 
information) , photography location information representing a 

20 place of photography, and subject information representing the 
type of a subject, in tag information TO of the image data sets 
SO at the times of photography thereof. 

The personal computer 10 has been installed with viewer 
software for viewing the image data sets SO, and a 

25 general-purpose Web browser. The user 1 uses the viewer 
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software for viewing the image data sets SO obtained by 
photography with the digital camera 11. The user 1 can also 
access the image storage server 2 by using the Web browser and 
can request classification of the image data sets SO from the 
5 image storage server 2 . When the user 1 requests classification 
of the image data sets SO from the image storage server 2, the 
image data sets SO are sent to the image storage server 2 via 
the network 3. At this time, user information for identifying 
the user 1 is also sent to the image storage server 2. The user 

10 information may be described in the tag information TO of the 
image data sets SO. 

The image storage server 2 comprises a storage database 
21 such as a hard disc or a RAID, an information database 22 
(not shown) , image organizing means 4 according to the first 

15 embodiment, a Web server 23, and thumbnail image generation 
means 24. The storage database 21 stores the image data sets 
SO, thumbnail image data sets STO, and classification 
information BO representing a result of classification of the 
image data sets SO, as will be explained later. The information 

20 database 22 stores various kinds of information such as the user 
information. The Web server 23 enables the user 1 to view the 
data sets and the information stored in the storage database 
21 via the network 3. The thumbnail image generation means 24 
generates the thumbnail image data sets STO representing 

25 thumbnail images from images represented by the image data sets 
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so. 

In the first embodiment, the storage database 21 stores 
the image data sets SO, the thumbnail image data sets STO, and 
the classification information BO. The image storage server 
5 2 may carry out procedures of the means thereof by using software 
having the procedures, instead of having the means themselves. 

The image organizing means 4 comprises image data input 
means 41, tag information reading means 42, information item 
specification means 43, and image classification means 44 . The 

10 image data input means 41 receives an input of the image data 
sets SO. The tag information reading means 42 reads the tag 
information TO added to the image data sets SO. The information 
item specification means 43 specifies information items to be 
used for classification of the image data sets SO from items 

15 in the tag information TO, and specifies priority among the 
specified information items. The image classification means 
44 classifies the image data sets SO into groups having a 
hierarchical structure of the specified information items 
according to the specified priority, and generates the 

20 classification information BO representing the result of 
classification . 

The image data input means 41 comprises a communication 
interface for receiving the image data sets SO sent from the 
personal computer 10. 

25 The image organizing means 4 selects the information 
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items to be used for image classification from the information 
items described in the tag information TO, and specifies the 
priority thereof . The image organizing means 4 also classifies 
the image data sets SO into the groups having the hierarchical 
5 structure of the specified information items, according to the 
priority that has been specified. 

Figure 2 shows diagrams for explaining classification of 
the image data sets SO. The tag information TO of each of the 
image data sets SO has the information items comprising the 

10 date/month/year of photography, the photography location, the 
subject type, and the name of the subject. These information 
items may be described automatically by the digital camera 11 
or by a manual operation carried out by the user 1 . In addition 
to the predetermined information items, the user 1 can 

15 arbitrarily generate an information item so that the user 1 can 
describe specific information in the item. For example, if the 
image data sets SO represent images of cooking, the user 1 may 
generate an item "ingredients" as one of the information items 
in the tag information TO so that the user 1 can describe "beef" 

20 as the content of the newly generated information item, for 
example. 

Therefore, the image classification means 44 classifies 
the image data sets SO into the groups having the hierarchical 
structure of the specified information items, according to the 
25 priority specified by the information item specification means 
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43. For example, in the case where the information items and 
the priority thereof are specified as "1. photography location, 

2. photography year, 3. photography month, and 4. photography 
date", the image data sets SO are first classified according 

5 to the photography location. Thereafter, the image data sets 
SO are further classified according to the photography year, 
and then by the photography month followed by the photography 
date. More specifically, folders having the hierarchical 
structure of "'photography location", "photography year", 

10 "photography month" and "photography date" are generated, and 
the image data sets SO are stored in the folders at the lowest 
level of the hierarchical structure. 

The image classification means 44 classifies the image 
data sets SO in the above manner, and generates the 

15 classification information BO representing the classification 
result . By referring to the classification information BO, the 
image data sets SO can be classified according to the specified 
information items . 

Figure 3 shows the result of classification. Figure 3 

20 shows the case where the information items and the priority 
thereof are "1. photography location, 2. photography year, 3. 
photography month, and 4 . photography date" . As shown in Figure 

3, a folder "year 2002" is generated under a folder "Disneyland". 
Under the folder "Year 2002" is generated a folder "May", and 

25 a folder "io th " is generated under the "May" folder. Folders 
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having the hierarchical structures such as "Hakone" - "Year 
2002" ~ "May" - "11 th ", and "My Parents' House" - "Year 2002" 
- "May" - «nth« are also generated. 

Figure 4 shows another result of the classification in 
5 the case where the information items and the priority thereof 
are "1. subject type, 2. photography year, and 3. photography 
month". As shown in Figure 4, a folder "People" is generated, 
and a folder "Year 2002" is generated under the "People" folder. 
Under the "Year 2002" folder is generated a folder "May". In 

10 addition, folders having a hierarchical structure such as 
"Mountain" - "Year 2002" - "May" are also generated. 

In the case where the classification result is displayed, 
the image data sets SO are shown as the thumbnail image data 
sets ST0 with reference to the classification information B0. 

15 The operation of the first embodiment will be explained 

next. Figure 5 is a flow chart showing procedures carried out 
in the first embodiment. 

First, the image data sets SO obtained by the user 1 with 
the digital camera 11 are sent from the personal computer 10 

20 of the user 1 to the image storage server 2 via the network 3, 
and the image storage server 2 receives the image data sets SO 
by using the image data input means 41 (Step SI) . The image 
data sets SO are stored in the storage database 21 (Step S2), 
and the thumbnail image generation means 24 generates the 

25 thumbnail image data sets ST0 from the image data sets SO (Step 
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S3) . The thumbnail image data sets STO are also stored in the 
storage database 21 (Step S4) . 

The tag information reading means 42 reads the tag 
information TO added to the image data sets SO (Step S5) , and 
5 the tag information is input to the image classification means 
44 . The user 1 specifies the information items and the priority 
thereof to be used for image classification, by using the 
information item specification means 43 (Step S6) . The image 
classification means 44 classifies the image data sets SO into 

10 the groups having the hierarchical structure of the specified 
information items, according to the specified priority, and 
generates the classification information BO (Step S7) . The 
classification information BO is stored in the information 
database 21 (Step S8) to end the process. 

15 The order of the procedures at Step S2, at Steps S3 and 

S4, and at Steps S5 to S8 is arbitrary, and the procedures may 
be carried out in parallel. 

The user 1 accesses the Web server 23 from the personal 
computer 10, and inputs an instruction to display the 

20 classification result of the image data sets SO he/she sent. 
The image data sets SO are then displayed on the personal 
computer 10 with reference to the classification information 
BO. 

Figure 6 shows the classification result of the image data 
25 sets SO displayed on the personal computer 10. As shown in 
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Figure 6, the classification result is shown as a classification 
result display screen 50 on the personal computer 10. In a left 
frame 50L on the left side of the screen 50 is shown the folder 
structure of the classification result, and in a right frame 
5 50R on the right side are shown the thumbnail images as 
representatives of the images classified in the respective 
folders. The user 1 can display the thumbnail images of the 
image data sets SO classified in each of the folders, by clicking 
a corresponding one of the folders in the right frame 50R of 

10 the screen 50. In Figure 6, the thumbnail image data sets ST0 
of the image data sets SO classified in the folder structure 
"Hakone" - "Year 2002" - "May" - "11 th " are shown. 

Furthermore, the name of each of the folders is also 
displayed as a title thereof. 

15 In the case where a portion of the image data sets SO is 

misclassif ied due to an error in description in the tag 
information TO or the like, the classification result needs to 
be changed. In this case, the user 1 can change the folder in 
which each of the misclassif ied image data sets SO is to be 

20 classified, by changing the folder in which the corresponding 
thumbnail image data set ST0 of the image data set SO is currently 
classified, with use of the personal computer 10. More 
specifically, the corresponding thumbnail image is dragged and 
dropped to a desired one of the folders. Consequently, the 

25 corresponding image data set SO can also be moved to the folder. 
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